Rubric-based Automated Japanese Short-answer Scoring and Support System Applied to QALab-3

We have been developing an automated Japanese shortanswer scoring and support machine for the new National Center written test exams. Our approach is based on the fact that accurately recognizing textual entailment and/or synonymy has been almost impossible. The system generates automated scores on the basis of evaluation criteria or rubrics, and human raters revise them. The system determines semantic similarity between the model answers and the actual written answers as well as a certain degree of semantic identity and implication. An experimental prototype operates as a web system on a Linux computer. To evaluate the performance, we applied the method to the second round of entrance examinations given by the University of Tokyo. We compared human scores with the automated scores for a case in which 20 allotment points were placed in five test issues of a world-history test as part of a trial examination. The differences between the scores were within 3 points for 16 of 20 data provided by the NTCIR QALab-3 task office.